Resolves GOL-449

Implement Oplog epoch fencing: oplog_metadata, epoch-checked appends, ShardLost relinquish

Ticket 4 made shard ownership a lease. A lease bounds how long a lost shard keeps being served, but it
cannot stop an executor that has already lost the shard from finishing a write it had started. An
executor frozen by a pause, a VM migration or a partition wakes up still believing it owns its agents.
Until now the only thing standing between its writes and the new owner's was the index_storage
primary key, which kills whichever executor happens to write second — often the rightful owner.

This PR records, per oplog, the shard epoch allowed to write it, and checks every batch of entries
against that record inside the transaction that inserts them. An executor whose epoch is behind is
refused at the storage. The one agent it was writing is given up (ShardLost): nothing more is written
for it, and the worker service resumes it on the shard's owner. The executor keeps serving everything
else.

Builds on #3834. #3766 (revoke drain) lands after this and builds on ActiveAgents::relinquish_matching.

What this adds

The fence in storage

• An ownership record per oplog. oplog_metadata (namespace, key, epoch) in the PostgreSQL and
  SQLite indexed storage (002_oplog_metadata.sql). It is written before an oplog's first entry,
  removed before its entries, and only moves forward: the upsert is a compare-and-set
  (WHERE oplog_metadata.epoch <= EXCLUDED.epoch).
• Every append asserts the epoch in the transaction that inserts it. PostgreSQL reads the record
  with SELECT ... FOR UPDATE; SQLite relies on its single-connection write pool (the code says what
  raising that cap would require). A missing record refuses too, and a refused batch leaves nothing
  behind. SQLite and multi-SQLite get their own transactional append_many.
• The epoch travels on the storage abstraction. IndexedStorage::append/append_many take
  shard_epoch: Option<ShardEpoch>. None asserts nothing: single-shard mode, the debugging service,
  a fork into a remote target, ephemeral oplogs and the archive layers. Backends that cannot fence say
  so through supports_epoch_fencing.

A refused write gives up one agent

• An oplog asserts the epoch its executor held when it opened it, claimed before the last index is
  read, so an executor that already lost the shard is refused before it replays.

• The refusal latches. IndexedStorageError::Fenced → OplogError::Fenced →
  WorkerExecutorError::OplogFenced → TrapType::Interrupt(InterruptKind::ShardLost). After the first
  refusal every add, pair, start and commit on that handle is refused without asking the storage.
  Entries buffered before the refusal stay readable, so a reader that took the index earlier never
  finds a gap.

• Refused writes are reported, never acknowledged. Oplog and the Worker commit APIs
  (commit_oplog_and_update_state, add_and_commit_oplog) return the fence. Callers propagate it, so
  nothing is done for an entry the storage refused:

  • AgentInvocationFinished is committed before a result is published to waiters;
  • an invocation is accepted only after its PendingAgentInvocation committed (unary and streaming);
  • a remote transaction's pre-commit/pre-rollback marker, a custom durable Start and the replay
    Jumps gate the side effect they precede;
  • cancel, revert, update and plugin (de)activation are not acknowledged;
  • an in-function retry whose Error entry is refused traps instead of retrying the effect.

  Transient storage failures keep panicking exactly as before; only the fence changes behaviour.

• Given up, never restarted in place. The agent is stopped without writing to its oplog or status
  and removed from ActiveAgents (scoped to its generation). OplogFenced crosses the wire as
  ShardingNotReady, which the worker service answers by refreshing its routing table and retrying on
  the owner. Pending invocations of a given-up agent get that error and no cached result, because
  the new owner runs them.

• Revoked and narrowed shards give their agents up instead of restarting them with
  InterruptKind::Restart, which would reopen the oplog at a stale epoch.

• The invocation loop never waits for a stop that waits for it. Once an agent is given up the loop
  stops picking up work, never creates a new instance, and writes no lifecycle, failure, update or
  streaming-session record for it. Loop-side cancel and manual-update enqueue refuse instead of waiting
  when the worker is stopping.

• A sweep never stalls lease renewal. Sweeps triggered by the renewal loop run on a single-flight
  task, so a slow relinquish (for example of an agent still loading) cannot let the lease lapse.
  AssignShards and startup still await the sweep.

Keeping epochs meaningful

• The shard manager repairs lost history automatically. A renewal claim above the recorded epoch
  raises the manager's floor. A re-registration after LeaseNotFound carries the shards the executor
  held (RegisterRequest.previous_shard_epochs). Executors report the epochs their refused writes saw
  (RenewShardLeaseRequest.fenced_shard_epochs), and the manager mints above them. Out-of-range epochs
  from the wire are ignored rather than overflowing.
• AgentInvocationStarted records the epoch that wrote it (shard_epoch: Option<u64>, raw oplog
  only; forks clear it).
• A routing miss before acceptance is retried, not surfaced. The new
  INVOCATION_REJECTION_REASON_SHARDING_NOT_READY carries InvalidShardId, ShardingNotReady and
  OplogFenced; the worker service retries it on the unary and streaming paths, and agent RPC maps it
  the same way. Before, InvalidShardId fell through to Internal, a terminal error.
• Startup refuses storage that cannot fence under a real shard manager
  (ShardManagerService::requires_oplog_fencing has no default on purpose).
• Fences and give-ups are logged with the agent and both epochs.

Supporting changes

• The test framework can pause and resume a spawned executor (SIGSTOP/SIGCONT): it looks dead to the
  cluster while it still believes it owns its shards, the case this PR is about.
• Docs: the deployment guide describes the fence, the backends that enforce it, the automatic repair
  and the rollout order; the persistence page lists PostgreSQL and SQLite as indexed storage.
  worker-executor-walkthrough.html and the understanding-durable-execution skill describe ShardLost,
  the fence and fallible commits.
• local-run/start.sh runs the executor and the debugging service on SQLite indexed storage.

Operational change and rollout

• An executor configured with Redis or in-memory indexed storage and a shard manager refuses to
  start, naming the setting to change (GOLEM__INDEXED_STORAGE__TYPE: Postgres, Sqlite,
  KVStoreSqlite, MultiSqlite or KVStoreMultiSqlite). The shipped default is still KVStoreRedis,
  so a distributed deployment on defaults fails at boot; changing the default belongs to ticket 6.
• Upgrade the shard manager and worker services before the executors. An upgraded executor reports
  a routing miss with the new rejection reason, which only an upgraded worker service retries, and sends
  epoch evidence in fields an older shard manager ignores.
• Roll forward only once any executor runs this release: migration 002 adds the table every
  append checks.

Also fixed along the way

Found by the audits; each has a test.

• Two writers at one epoch. A shard manager whose state was wiped mints from zero again and could
  hand a live owner's epoch to another executor; both passed the fence, because the record held only a
  number. The record now names the writing process as well, an equal epoch from anyone else is refused,
  and the refusal is reported so the manager mints past it. The writer is a per-process identity, not
  the executor's lease identity, which is regenerated on LeaseNotFound and would fence a process off
  its own agents after every re-registration.
• Epoch overflow, end to end. A negative value in the epoch column read back through an unchecked
  cast as a near-u64::MAX epoch, was reported as evidence, and walked the manager's own mint to the
  ceiling, where it panicked. The reads are checked, the mint is fallible - a shard whose epoch cannot
  advance stays where it is instead of aborting the manager - and LeaseEpoch gained the checked
  advance its shard-side twin already had.
• The WebSocket session API now reports the new rejection reason as its own retriable code rather
  than InternalError, so a session client reroutes like every other caller.
• Quota leases: a RenewQuotaLease/ReleaseQuotaLease call with epoch u64::MAX panicked the
  shard manager. It is now an ordinary stale epoch.
• Archive transfers: a byte-identical repeat of an already-stored compressed chunk is accepted
  instead of panicking on the key conflict.

Known gaps

Every open risk from the review was re-verified against this head. These are what is left.

Handed to #3766, which rewrites the revoke path this sits in:

• suspend_worker still flushes the status blob for an agent relinquished by a revoke, and status
  writes carry no epoch, so a stale one can overwrite the new owner's.
• drop_prefix is not epoch-checked, so a transfer that outlives its ownership can still trim or
  delete an oplog the new owner holds.
• Revoke keeps this PR's relinquish sweep but skips upstream's owner-retirement steps: the producer
  fence, stopping the attachment reconciler, draining the lifecycle, and begin_delete on the status
  flusher and checkpointer. No durability impact - late writes are replayed or refused - but the
  liveness gaps are real.

Also for #3766: Worker::relinquish now returns without its own stop when a deletion already owns
the retirement, so the "relinquish awaits loop exit" contract no longer holds on that branch.

Accepted here:

• The fence covers the oplog only. Key-value records, the status blob and blob payloads carry no
  epoch. The oplog - the only thing replay reads - stays correct; a stale status can be observed until
  the agent's next status change on its new owner.
• Ephemeral oplogs are not fenced, by design: they are never replayed.
• External effects. A call already in flight when the shard moves can happen twice, the same
  at-least-once exposure a crash has. A call that has not started is now refused before the effect
  runs, including for writes marked idempotent, which previously checked nothing until the commit
  after the effect.
• Rollout is roll-forward only, and the shard manager and worker services go first. See
  deployment.
• The shipped default is still KVStoreRedis, which cannot fence, so a distributed deployment on
  unmodified defaults fails at startup until the setting is changed. Changing the default is ticket 6.
• The worker service's routing retry loop has no terminal case (pre-existing, untouched here): on
  exhaustion it computes a backoff, never awaits it, resets the counter and loops. This PR routes a new
  and more frequent failure into it, so a fence that does not resolve presents as a request that never
  finishes. Worth fixing deliberately rather than inside a fence PR.

Resolved with evidence, not left open:

• A push lost when a handler is cancelled mid-persist (flagged uncertain in review): the durable-stream
  write runs on a spawned actor task and is queued before the caller's only await point, and the effects
  guard poisons the producer rather than reporting success. Not a defect.
• A stale trailing compressed chunk: pre-existing upstream behaviour - abort_transfer_in_drop is
  older than this branch - and the orphan is swept by the next drop_prefix.

How it is tested

• Storage: the compare-and-set, the fenced append on stale, ahead and missing records, delete,
  None bypassing the check, and that a refused batch leaves nothing behind, on every backend
  (non-fencing backends must accept the same writes). Deliberately breaking each backend's check turns
  the suite red.
• Oplog service over a SQLite store that fences: an owning epoch writes; a stale epoch is refused at
  its first write and stays refused; once latched, adds below the commit threshold are refused and
  earlier indices stay readable; a writer holding a deleted oplog is refused; an indeterminate write on
  a moved shard is reported as fenced; a fenced cached handle is never handed out again.
• Executor: a fenced write escaping a host call classifies as ShardLost and a transient storage
  failure does not; every lost-shard exit path gives the agent up; a slow sweep does not stall renewal
  and concurrent announcements coalesce; an invocation enqueued onto a fenced oplog is refused before
  acceptance with SHARDING_NOT_READY; a caller waiting on an invocation fenced inside a host call is
  told to reroute, and the executor writes nothing after it.
• The writer column: on every backend, a second process presenting an epoch the record already
  holds is refused and the refusal names the writer as the reason; the same process re-opening at the
  same epoch is accepted, which is what happens on every cache eviction; and a newcomer minted above
  the collision takes the oplog over, after which the old owner is the one refused. On the manager
  side, a fence reported by a shard's own assignee at its own epoch is minted past, while the same
  report from anyone else is the ordinary loser of a shard move and moves nothing.
• Loop-side guarantees that previously had no deterministic test: a stop racing a manual update
  finishes both ways round rather than deadlocking; a retry scheduled before a revoke does not resume
  an agent this executor gave up; a start through a relinquished generation is refused and leaves the
  generation that replaced it alone; a foreign stream mapping on a fenced oplog reports the fence,
  commits nothing and releases the session lock.
• End to end
  (an_executor_paused_until_its_shards_move_cannot_finish_the_invocations_it_started): eight agents
  each start an in-flight invocation; three of four executors are frozen until their shards move; the
  fourth takes the shards over and finishes the work; the frozen ones are thawed. Each invocation
  returns exactly once, each agent gains exactly one AgentInvocationFinished for the method, the
  stored owning epoch has moved, and every executor still serves afterwards. With the PostgreSQL append
  check disabled the test fails: a thawed executor's stale append collides with the owner's and aborts it.

🤖 Generated with Claude Code